Programming languages are tools. Just like hammers. There's nothing personal about them. They just help you build stuff. Yet, there are many religious wars about programming languages out there. Wars which despite being all about science, are much more related to emotions, beliefs and personal aggressions than about objective arguments.

This is funny because do you know of any religious wars about hammers? So what's the difference?

Here's a recent personal example. A guy explaining how the static typing of Common Lisp works (type declarations) and what kind of performance-oriented optimization can be achieved with it (C-like, static but weak typing in SBCL for instance). And then, there's inevitably the troll (whom I know knows better) in the audience who goes:

So, yeah, what you're doing is just C code, and you have to type manually, and it's ugly. So if it's just for writing C code, I don't see the point in using another language.

Hmmm. Let's see. So, yes indeed, static typing in Common Lisp is ugly. And yes, it's not even strong typing. And yes, it would be nicer to have run-time hotspot detection and automatic type-dependant optimization like what's found in some other languages or virtual machines, rather than having to do it by hand (BTW, that would make for a nice Ph.D. I think). But what does that really tell you? That no language is perfect? Wow, thank you very much, that's new. For as much as I think that Lisp The Idea™ is perfect, I don't think anyone ever pretended that Common Lisp (or any Lisp for that matter) was. But is that a reason for not using it at all and sticking to C? Any sane computer scientist knows that the choice of a language doesn't boil down to only one parameter. Any computer scientist who tells otherwise is a troll.

In spite of all its defects, I still have a gazillon reasons to prefer Common Lisp over C, C++, or any language that I know of currently (and this may very well change in the future). The flexibility of lambda lists, the macros, the MOP or more generally its structural and behavioral reflexivity. Its run-time compilation, debugging, introspection and intersession capabilities. These are just a few examples. Still, I don't deny anyone the right to prefer another language for whatever purpose and whatever reasons they may feel legitimate.

So, I normally just ignore those purposedly trollesque and completely idiotic remarks. Yet, sometimes like yesterday, I snap. It gets on my nerves and I become all upset and angry. Why? I never get angry when someone tells me that my hammer is a piece of crap (it's not). I do enough Aikido to know how to control my temper, but for some reason, it doesn't always work when it comes to programming languages. So what is it about them that in spite of all your efforts, you can't help from getting personally and emotionally involved once in a while?

I think the answer becomes apparent when you consider the artistic aspect in programming. When an artist creates a piece (music, theater, dance, painting, architecture, whatever you like), (s)he exposes a very intimate part of himself through his creation. The art "is" the artist, and the artist "is" his art. In doing so, he puts himself in danger. It's like yelling out to the whole world Hey, look, I'm vulnerable right here!!. It's a well know fact that many artists are very fragile, in the sense that they suffer from their creation not being liked. Because a piece of art is intrinsically a piece of the artist himself, when you say I don't like this piece of art, you're actually saying I don't like the artist. Then, it's up to artist to handle the fact of not being liked.

And that's the whole problem, which, as a musician, I know all too well. Where does the artistic fiber come from? It's an urge to express yourself. To express something that you can't express in any other way. A very deep and perpetual wound of some sort, a feeling of not really belonging. More importantly, it's a calling. Sometimes, the simple fact of creating is enough to heal you a bit, but more often, you create in order for your creation to be seen or heard. So yes, it's a calling to the Others. You expect them to answer your call by telling you that they like you (your art, but that is the same). Artists often have this urge to be liked by the Others. So when you dislike some artwork, you're also not liking the artist himself (the part of him that lives in his creation) and you're actually giving him the exact opposite of what he was looking for. And that hurts.

Back to programming languages. Why do we get all emotional about them, and not about hammers? The answer is in fact quite simple. Look at an architectural masterpiece. Do you see the hammer that was used to build it? Now look at a software masterpiece. Do you see the language that was used to write it? That's the crucial difference. You cannot decouple the language from the software, even once it has been written (the art is not in the executable; it's in the source code). The language itself will always be here for you to contemplate.

All in all, I think that's why there will always be language wars. Languages are not just tools, actually. They're not just like hammers. As soon as you care about the code you write, your software becomes artwork, you become an artist, and you start to be personally and emotionally involved. Your software becomes part of you. And contrary to the hammer, your sticky programming language, being intrinsically bound to the artwork, also becomes part of you. That's when the battle for objectivity is lost. By criticizing the language, the troll also criticizes your artwork, and in doing so, he tells you that he doesn't like you. That may hurt.

It's good to consider programming as art. Unfortunately, this also means that there will always be language wars.

A new version of Clon, the Command-Line Options Nuker is out.

The most important change in this release is the support for LispWorks, which brings the number of supported implementations to 8. One left to go, and I may eventually switch to RC status. Thanks to Martin Simmons for providing a fully functionnal version of LW 6.1. As for CLISP and Allegro, there is an optional dependency on CFFI for LispWorks.

Two backward incompatible changes that may affect you:

• Variables renamings: *current-context* has been renamed *context*, and *default-synopsis* has been renamed *synopsis*. This should remain transparent unless you're using Clon in a somewhat advanced way.
• clon:exit has been upgraded to SBCL's new quitting protocol. If you use this function (or if you want to compile the demo programs), please upgrade to SBCL 1.0.57.

Finally, support for terminal autodetection and stream handling in general has been improved for all implementations.

Grab it at the usual place.

I'm happy to announce that my contribution to TUG 2012, the next TeX Users Group International conference, has been accepted. Please find the title and abstract below.

Star TeX, the Next Generation

In 2010, I asked Donald Knuth why he chose to design and implement TeX as a macro-expansion system (as opposed to more traditional procedure calls). His answer was that:

1. he wanted something relatively simple for his secretary who was not a computer scientist,
2. the very limited computing resources at that time practically mandated the use of something much lighter than a true programming language.

The first part of the answer left me with a slight feeling of skepticism. It remains to be seen that TeX is simple to use, and when or where it is, its underlying implementation has hardly anything to do with it.

The second part of the answer, on the other hand, was both very convincing and arguably now obsolete as well. Time has passed and the situation today is very different from what it was 50 years ago. The available computing power has grown exponentially, and so has our overall skills in language design and implementation.

Several ideas on how to modernize TeX already exist. Some have been actually implemented. In this talk, I will present mine. Interestingly enough, it seems to me that modernizing TeX can start with grounding it in an old yet very modern programming language: Common Lisp. I will present the key features that make this language particularly well suited to the task, emphasizing on points such as extensibility, scriptability and multi-paradigm programming. The presentation will include reflections about the software engineering aspects (internals), as well as about the surface layer of TeX itself. Most notably, I will explore the possibilities of providing a more consistent syntax to the TeX API, while maintaining backward compatibility with the existing code base.

I'm happy to announce a new release of Clon, the Command-Line Options Nuker for standalone Common Lisp executables. In addition to a lot of uninteresting code and infrastructure changes, this new release comes with several important improvements and new features.

At the end-user level

• there is a new error handler available via the --clon-error-handler option, called "interactive". This error handler provides the same restarts as the one called "none" (which actually triggers the Lisp debugger), but in a less frightening way for end-users not knowing about Lisp at all. In particular, the error and restart messages are more readable and you don't see a Lisp stack anywhere. See the end-user manual and the user manual for more information.
• there is a new option called --clon-lisp-information which, as its name suggests, provides information about the underlying Lisp (implementation type and version). This will in fact be more useful for developers than for end-users, but it's still and end-user level feature. See the end-user manual for not much more information.

At the user-level

• Clon now provides a command-line polling API through the functions cmdline-options-p and cmdline-p. See the user manual for more information.
• Support for using Clon interactively, that is, without dumping executables has been improved. This is mostly useful for debugging purposes. See the user manual for more information.
• Clon now provides a compile-time / run-time unified configuration facility thanks to a variable named cl-user::com.dvlsoft.clon.configuration that is handled before the ASDF system is loaded. Thanks to this, Clon is now able to communicate its own indentation information to (X)Emacs directly (thanks to a process that I've previously described here), and also handles portability problems in a smoother way (see below).
• One of the available configuration options is called :restricted mode. In this mode, Clon never attempts to communicate with ttys via ioctl calls, at the expense of terminal autodetection (size and highlighting). This is implemented by making a termio ASDF module conditionally loaded in the system definition. There are cases where Clon will switch to restricted mode automatically (e.g. when using CLISP compiled without FFI support). However, some other situations are more problematic, for instance when using SBCL under MinGW, in which case the SB-GROVEL module is available but doesn't work. In such situations, it is necessary to configure Clon explicitely for restricted mode before loading the system. See the user manual for more information.

That's it. Grab it at the usual place. Yesterday, I realized that it's been slightly more than a year since the b19 release. Gee, time flies like the wind...

Here's yet another indentation hack that I came up with recently.

All the work done by Nikodemus on the Slime indentation contrib is pretty cool, especially the notion of indentation style (though I wish the styles were Custom variables, but that is another story). I tend to use indentation styles for global, maybe collaborative preferences, but on several occasions however, I find that this approach has a couple of drawbacks.

• One of them is that the indentation information is far away from the corresponding symbol, in a separate file. If you change a function's prototype for instance, you may also need to load the file(s) in which the corresponding style(s) is (are) defined and edit them.
• The other problem is that if you want to let other people edit your source code and honor your indentation style, you also need to provide them with the style definition, and they need to load it separately.

For those reasons, I tend to think that the indentation style approach is not very well suited for project-specific indentation settings. What I would like is to provide indentation information close to the function definition, and also to have that information automatically available when anyone loads the project into Slime. Here's a way to do it.

The key to success here is the function swank:eval-in-emacs which, as its name suggests, sends some Emacs Lisp code to your (X)Emacs session for evaluation. This function effectively allows you to trigger some Emacs Lisp computation from a Common Lisp file. Remember that indentation information is stored in the common-lisp-indent-function property of a symbol. The function clindent below does this:

(defun clindent (symbol indent)
  "Set SYMBOL's indentation to INDENT in (X)Emacs.
This function sets SYMBOL's common-lisp-indent-function property.
If INDENT is a symbol, use its indentation definition.
Otherwise, INDENT is considered as an indentation definition."
  (when (and (member :swank *features*)
	     (let ((configuration
		     (find-symbol "MY.PACKAGE.CONFIGURATION" :cl-user)))
	       (when (and configuration (boundp configuration))
		 (getf (symbol-value configuration) :swank-eval-in-emacs))))
    (funcall (intern "EVAL-IN-EMACS" :swank)
	     `(put ',symbol 'common-lisp-indent-function
		   ,(if (symbolp indent)
			`(get ',indent 'common-lisp-indent-function)
		      `',indent))
	     t)))

As explained in the docstring, this function will ask (X)Emacs to put SYMBOL's common-lisp-indent-function property to a definition, either provided directly, or retrieved from another symbol. For example, if your package defines an econd macro, you may want to call it like this:

(clindent 'econd 'cond)

This function ensures that Swank is actually available before using it (first condition in the and clause). I will explain the other weird bits later on.

The next question is when exactly do we want to call this function? The answer is: pretty much on all occasions. Your code might be loaded from source and interpreted, or it might be compiled. But then, it might be compiled within or outside a Slime environment. In any case, you want your indentation information to be sent to (X)Emacs everytime it's possible. So obviously, we're gonna wrap this function in an eval-when form thanks to a macro. This is also a good opportunity to save some quoting.

(defmacro defindent (symbol indent)
  "Set SYMBOL's indentation to INDENT in (X)Emacs.
SYMBOL and INDENT need not be quoted.
See CLINDENT for more information."
  `(eval-when (:compile-toplevel :execute :load-toplevel)
     (clindent ',symbol ',indent)))

And now, right on top of your econd definition, you can just say this:

(defindent econd cond)

Now here's one final step. If your package uses its own readtable, it's even more convenient to define a reader-macro for indentation information. I choose #i:

(defun i-reader (stream subchar arg)
  "Read an argument list for the DEFINDENT macro."
  (declare (ignore subchar arg))
  (cons 'defindent (read stream)))
 
(set-dispatch-macro-character #\# #\i #'i-reader *readtable*)

And now, the code in my package will look like this:

#i(econd cond)
(defmacro econd #|...|#)

Pretty cool, eh?

All right. We still have two weirdos to explain in the clindent function.

First, you noticed that the function's computation is conditionalized on the existence of a cl-user::my.package.configuration variable, which actually stores a property list of various compiling or loading options for this package. The option we're interested in is :swank-eval-in-emacs, which must be set to non-nil. Here's why. The execution of Emacs Lisp code from Swank is (rightfully) considered as a security risk so it is disabled by default. If you want to authorize that, you need to set the (Emacs) variable slime-enable-evaluate-in-emacs to t. Otherwise, calling swank:evaluate-in-emacs is like calling 911. So we have a chicken-and-egg problem here: if we want to avoid an error in clindent, we would need to check the value of this variable, but in order to do that, we would need to evaluate something in (X)Emacs ;-)

The solution I choose is hence to disable the functionality by default, and document the fact that if people want to use my indentation information, they need to set both the Slime variable and my package-specific option to non-nil before loading the package (possibly setting them back to nil afterwards). They also need to trust that I'm not going to inject anything suspicious into their (X)Emacs session at the same time...

The last bit we need to explain is the final t argument passed to swank:eval-in-emacs. The corresponding parameter is called nowait in the function's prototype. It has something to do with asynchronous computation, and in fact, I don't really know what's going on under the hood, but what I do know is that if you set it to t, Swank doesn't care about the return value of your form anymore, which is fine because we're only doing a side effect. On the other hand, if you omit that parameter, Swank will try to interpret the return value in some way, and you will most probably get a serialization error. Indeed, the return value is the indentation definition itself, so for example, (&rest (&whole 2 &rest 1)) doesn't make (Common Lisp) sense.

That's it. Happy indenting!

Sometimes, I miss the string concatenation capability of C. I mean, the way that you can split a long string into smaller ones and have them automatically concatenated together. The format function has a tilde (~) directive that does something along these lines, but there are two problems:

• first, I'm not necessarily facing the problem in format calls,
• next, my favorite text editor cannot properly indent the string contents when I press the TAB key.

Here's an example:

(defclass sample ()
  ((slot :documentation
	 "A very long slot documentation, that doesn't even fit in 80 columns, which is a shame...")))

If that were C code, I could write this:

(defclass sample ()
  ((slot :documentation
	 "A very long slot documentation, "
	 "that doesn't even fit in 80 columns, "
	 "which is a shame...")))

But (thank God) this is not C code. We can come very close to that however. Here's a small reader function that will automatically concatenate strings prefixed with a tilde (~) character (in honor of format's directive):

(defun tilde-reader (stream char)
  "Read a series of ~\"string\" to be concatenated together."
  (declare (ignore char))
  (flet ((read-string (&aux (string (read stream t nil t)))
	   (check-type string string "a string")
	   string))
    (apply #'concatenate 'string
	   (read-string)
	   (loop :while (char= (peek-char t stream nil nil t) #\~)
		 :do (read-char stream t nil t)
		 :collect (read-string)))))

Let's add it to the current readtable:

(set-macro-character #\~ #'tilde-reader)

And now I can write this:

(defclass sample ()
  ((slot :documentation
	 ~"A very long slot documentation, "
	 ~"that doesn't even fit in 80 columns, "
	 ~"which is a shame...")))

Everyday of my life I thank Common Lisp for being so flexible. The next step would be to make cl-indent.el aware that tilde-strings really are the same object, and hence should be vertically aligned in all contexts. I'm not there yet. Jeeze, this indentation hacking will never end... :-)

UPDATE: since the original publication of this blog entry, Nikodemus Siivola and I have done some more work on various other aspects of Common Lisp indentation, and I must say that the result is pretty satisfactory. Nikodemus has merged all the changes into Slime, and I have done so for XEmacs. If you're an XEmacs user, you don't need to use the slime-indentation contrib to get these improvements. Simply upgrade the "prog-modes" package and load cl-indent.el.

I have just modified XEmacs to improve the indentation of Common Lisp code. This change involves two things: the support for multiple method qualifiers in a call to defmethod and, much more importantly, a cleaner and more flexible scheme for indenting lambda lists. The patch has also been submitted to the GNU Emacs developers. Below is a more detailed description of what you get with these changes.

Method qualifiers

Until now, only one method qualifier was understood. Below are some examples demonstrating that the one method qualifier and the argument list are indented by 4 characters, and the method's body only by 2:

(defmethod foo :around ()        (defmethod foo :around
  do-this)                           ()
                                   do-this)
(defmethod foo
    :around ()                   (defmethod foo
  do-this)                           :around
                                     ()
                                   do-this)

Now let's add a second method qualifier:

(defmethod foo comb :around ()        (defmethod foo	    
	   do-this)			  comb :around () 
					  do-this)

Woops. Neither is correct. But now, you get this instead:

(defmethod foo comb :around ()        (defmethod foo	    
  do-this)				  comb :around () 
					do-this)

Three more examples to show how confused we were:

(defmethod foo comb :around     (defmethod foo comb     (defmethod foo 
  ()				    :around		    comb       
  do-this)			  ()			    :around    
				  do-this)		  ()	       
							  do-this)

And how better we just got:

(defmethod foo comb :around     (defmethod foo comb     (defmethod foo
    ()				    :around		    comb      
  do-this)			    ()			    :around   
				  do-this)		    ()	      
							  do-this)

Indeed, you can see that everything between the method's name and its body is now correctly indented by 4.

Lambda Lists

The next round of changes deals with the formatting of lambda-lists. As such, this will apply everywhere a lambda-list is expected, such as defun, defgeneric, defmethod etc. First of all, here are two examples showing that we were not very clever before:

(defun foo (mand1 mand2
	    &optional opt1
	    opt2
	    &rest args
	    &key key1 key2
	    (key3 val3) &aux aux1
	    aux2)
  do-this)

(defun foo (mand1 mand2 &optional opt1
	    opt2
	    &rest args
	    &key key1 key2
	    (key3 val3) &aux aux1
	    aux2)
  do-this)

Basically, everything was blindly aligned at the beginning of the lambda-list. Here is what you get now:

(defun foo (mand1 mand2
	    &optional opt1
	      opt2
	    &rest args
	    &key key1 key2
	      (key3 val3) &aux aux1
			    aux2)
  do-this)

(defun foo (mand1 mand2 &optional opt1
			  opt2
	    &rest args
	    &key key1 key2
	      (key3 val3) &aux aux1
			    aux2)
  do-this)

The difference is that keyword parameters are indented with respect to their corresponding keyword. The amount of indentation is provided by a new customizable user option named lisp-lambda-list-keyword-parameter-indentation (oh boy, what a mouthful). If you prefer, you can also have the parameters vertically aligned with each other. Set the new customizable user option named lisp-lambda-list-keyword-parameter-alignment to t and you will get this instead:

(defun foo (mand1 mand2
	    &optional opt1
		      opt2
	    &rest args
	    &key key1 key2
		 (key3 val3) &aux aux1
				  aux2)
  do-this)

(defun foo (mand1 mand2 &optional opt1
				  opt2
	    &rest args
	    &key key1 key2
		 (key3 val3) &aux aux1
				  aux2)
  do-this)

Finally, just as you could align keyword parameters together, you can also align the keywords together. Set the new customizable user option named lisp-lambda-list-keyword-alignment to t, and you will get this (only the second example differs):

(defun foo (mand1 mand2 &optional opt1
				  opt2
			&rest args
			&key key1 key2
			     (key3 val3) &aux aux1
					      aux2)
  do-this)

These are in fact my preferred settings, although both alignment options default to nil. Here is a final example demonstrating how you could format a long lambda-list with plenty of arguments:

(defun foo (mand1 mand2
	    &optional
	      opt1 opt2 opt3
	    &key
	      key1 key2 key3
	    &aux
	      aux1 aux2 aux3)
  do-this)

These indentation problems have been a huge peeve of mine for quite a long time. I hope you will find the changes useful!